Decoupling the Data Geometry from the Parameter Geometry for Stochastic Gradients

Large-scale learning problems require algorithms that scale benignly with respect to the size of the dataset and the number of parameters to be trained; leading numerous practitioners to favor the classic stochastic gradient descent (SGD [1, 2, 3]) over more sophisticated methods. Besides its fast convergence, SGD has been observed to sometimes lead to signi cantly better generalization performance than batch gradient descent. SGD is also quicker than batch methods in adapting to non-stationary data distributions. Its Achilles heel are the inherently sequential updates, making it very di cult to parallelize across many machines; which is clashing with the goals of large-scale learning. Our goals here are twofold. On the theoretical level, we want to gain a fuller understanding of how the dynamics of stochastic updates contrast with those of batch updates, and how they are a ected by the conditioning of energy surfaces, the presence of local optima, and the properties of the data distribution. On the practical level, we want to use this knowledge to design more e cient mini-batch SGD variants (which are parallelizable), together with robust settings for their hyper-parameters. The study of stochastic gradient methods dates back over six decades [1, 2, 3, 4, 5, 6, 7], but to our knowledge, the present questions remain understudied. The most similar viewpoint is found in approaches based on natural gradients and information geometry [8, 9, 10].